-
Notifications
You must be signed in to change notification settings - Fork 1.1k
[Parquet] perf: Create Utf8/BinaryViewArray directly rather than via ArrayData
#9121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| pub fn into_array(self, null_buffer: Option<Buffer>, data_type: &ArrowType) -> ArrayRef { | ||
| let len = self.views.len(); | ||
| let views = Buffer::from_vec(self.views); | ||
| let views = ScalarBuffer::from(self.views); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new formulation is simpler too, which is a nice side effect
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
run benchmark arrow_reader |
|
This appears to show reading StringView getting slower. I will try and reproduce |
|
🤖 |
This comment was marked as outdated.
This comment was marked as outdated.
scovich
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
|
🤔 looks like this maybe made things slower -- will investigate |
|
I could not reproduce the benchmark results locally However, note the number of outliers found I will see if the benchmark needs adjusting (perhaps with more rows, etc) |
This comment was marked as outdated.
This comment was marked as outdated.
1 similar comment
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
…b/arrow-rs into alamb/less_parquet_view_allocations
This comment was marked as outdated.
This comment was marked as outdated.
| let views = ScalarBuffer::from(self.views); | ||
| let nulls = null_buffer | ||
| .map(|b| NullBuffer::new(BooleanBuffer::new(b, 0, len))) | ||
| .filter(|n| n.null_count() != 0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
replicates the behavior of ArrayDataBuilder in
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
The benchmakrs now show a small but repeatable improvement for the array types. I am not sure how real it is but I am now convinced this PR is an improvement |
|
I ran the benchmark for "arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs" locally with |
|
Thank you for double checking @jhorstmann and for the reviews @jhorstmann and @scovich |
Which issue does this PR close?
make_array) #9061Rationale for this change
ArrayData(1% improvement) #9120Creating Arrays via ArrayData /
make_arrayhas overhead (at least 2 Vec allocations) compared to simply creating the arrays directlyViewArrays also have an extra Vec allocation (to hold their buffers)
What changes are included in this PR?
Update the parquet reader to create ViewArrays directly
Are these changes tested?
By CI
Are there any user-facing changes?